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The auditory mismatch responses are elicited in absence of directed attention but are 
thought to reflect attention modulating effects. Little is known however, if the deviants 
in a stream of standards are specifically directing attention across modalities and how 
they interact with other attention directing signals such as emotions. We applied the 
well-established paradigm of left- or right-lateralized deviant syllables within a dichotic 
listening design. In a simple target detection paradigm with lateralized visual stimuli, we 
hypothesized that responses to visual stimuli would be speeded after ignored auditory 
deviants on the same side. Moreover, stimuli with negative valence in the visual domain 
could be expected to reduce this effect due to attention capture for this emotion, resulting 
in speeded responses to visual stimuli even when attention was directed to the opposite 
side by the auditory deviant beforehand. Reaction times of 17 subjects confirmed the 
speeding of responses after deviant events. However, reduced facilitation was observed 
for positive targets at the left after incongruent deviants, i.e., at the right ear. In particular, 
significant interactions of valence and visual field and of valence and spatial congruency 
emerged. Pre-attentive auditory processing may modulate attention in a spatially selective 
way. However, negative valence processing in the right hemisphere may override this 
effect. Resource allocation such as spatial attention is regulated dynamically by multimodal 
and emotion information processing. 
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INTRODUCTION 

Spatial attention distribution serves to enable an efficient interac- 
tion with the environment. The importance of such a mechanism 
is reflected in the huge impact of impairments of attention 
in the left visual field — unilateral neglect — on simple activities 
of daily living in patients with right-hemisphere damage (for 
a review see Danckert and Ferber, 2006). Valid spatial cueing 
of behaviorally relevant target stimuli can facilitate reaction to 
these targets; invalid cueing can enhance reaction times (Posner, 
1980). Previous studies distinguished between exogenous and 
endogenous — stimulus-driven and goal-directed — attention (for 
a review see Bartolomeo and Chokron, 2002). Exogenous cues 
trigger stimulus-driven shifts of attention when appearing in the 
left or right hemispace, even if cue and target stimuli are presented 
in different modalities (e.g., Eimer and Driver, 2001). 

Another important factor concerning the distribution of atten- 
tion in space is the salience of stimuli. Emotional valence of stim- 
uli can influence attention-related processing (Eastwood et al, 
2001; Fenske and Eastwood, 2003). Negative valence seems to 
serve as an attention "magnet"; it can narrow the attention focus 
accompanied by enhanced processing of the negative stimuli 
(Fenske and Eastwood, 2003). In visual search, negative stimuli 



are found faster and with lesser errors than positive stimuli 
(Eastwood et al, 2001). Stimuli with negative valence can even 
overcome neglect symptoms in the left visual field in patients with 
right-hemisphere damage (Grabowska et al, 2011). On the other 
hand, it was shown that negative emotionally arousing stimuli 
interfere with processing of left visual field targets when compet- 
ing for attention resources (Hartikainen et al, 2007). A neural 
network of limbic regions like amygdala and insula with the ante- 
rior cingulate cortex is known to play a role in salience detection 
and evaluation (for a review see Seeley et al., 2007; Menon and 
Uddin, 2010; Santos et al, 2011). 

Deviant acoustic stimuli in a sequence of rapidly presented 
standard stimuli elicit automatic mismatch responses in auditory 
cortices measured by electrophysiological as well as neuroimaging 
methods (EEG: Naatanen et al, 1989; fMRI: Mathiak et al, 2002). 
In spite of the pre-attentive nature of these responses, stimulus- 
driven attention shifts to the side of the deviant are possible 
(Naatanen, 1995; Schroger, 1996). In one of few studies on behav- 
ioral equivalence of mismatch processes, Schroger (1996) used a 
dichotic listening design with two different stimulus streams on 
both ears and instructed participants to ignore one ear while they 
had to do a GoNogo-task on the other ear. It was demonstrated 
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that reaction time costs occurred if the target stimulus of the 
GoNogo-task was preceded by a deviant on the to-be-ignored ear. 
The authors concluded that deviants in an oddball design elicited 
a shift of attention within the auditory domain. 

The goal of the present study was to demonstrate spatial shifts 
in visual attention by lateralized auditory deviants in an unat- 
tended oddball design. Visual stimuli of positive and negative 
valence in the left or right visual field represented target stim- 
uli, which were either spatially congruent or incongruent with the 
deviating dichotic left-ear or right-ear stimuli. An attention shift 
to the side of the deviant was hypothesized, resulting in response 
acceleration to spatially congruent visual stimuli. Negative stimuli 
were expected to reduce reaction time costs in incongruent trials. 
Therefore, the ANOVA of reaction times was expected to reveal an 
effect of spatial congruency and more importantly an interaction 
between congruency and valence. 

MATERIALS AND METHODS 
SUBJECTS 

Seventeen healthy volunteers (11 females, age 19-43 years, mean 
age 27.5 ± 6.7 years) participated in the experiment. All subjects 
were right-handed as indicated by the laterality quotient of the 
Edinburgh Inventory (Oldfield, 1971). To exclude any psychiatric 
condition, participants were screened with the Structured Clinical 
Interview (SCID-I; Wittchen et al., 1997) for the Diagnostic and 
Statistical Manual of Mental Disorders (DSM-IV). The study was 
approved by the local Ethics Committee of the Medical School 
of the RWTH Aachen University and was conducted in accor- 
dance with the Code of Ethics of the World Medical Association 
(Declaration of Helsinki). Informed consent was obtained prior 
to participation in the study. 

STIMULI AND DESIGN 

In the mismatch design, deviant stimuli in a sequence of standard 
stimuli are known to be detected even in the absence of directed 
attention. Therefore, we employed an unattended sequence of 
stimuli. To obtain an effect on spatial attention, dichotic stim- 
uli were applied and deviance occurred on one side only. On this 
auditory "background" visual stimuli were presented with a sim- 
ple detection task. The schematic faces were presented shortly 
(175 ms) after either a standard or a deviant stimulus. To test spa- 
tial attention these were either at the left or at the right side and 
to test valence effects the schematic face bore either a positive or 
negative expression. 

Dichotic stimulation 

Processing of left- and right-ear stimuli usually interacts at early 
processing levels, e.g., due to early processing of inter-aural time 
and level differences. Therefore, we chose a paradigm, which has 
been shown to elicit side-specific responses at the level of the audi- 
tory cortices (Hertrich et al, 2002). The auditory stimulation was 
chosen to match the study in Schock et al. (2012), but conducted 
in an independent sample. Sequences of consonant-vowel (CV) 
syllables /ba/ and /ka/ were applied dichotically. The /ba/ was con- 
stituted of two different versions from the same speaker providing 
random phase-relation between left- and right-ear. The stimuli 
were always synchronous over both ears. The deviant events were 



composed of the syllable /ka/ presented to one ear and the stan- 
dard syllable to the other ear (/baL/-/ka/ or /ka/-/baR/). Thus, 
the two cues voice-onset time and place-of-articulation changed 
facilitating a robust mismatch response over subjects even when 
stimuli were ignored. The stimuli had the same duration (300 ms) 
and sound level. 

The task-irrelevant oddball sequence with dichotic stimuli 
was administered over noise-cancellation headphones (HDA 200, 
Sennheiser, Germany) with a constant stimulus onset asynchrony 
(SOA) of 500 ms. Sound pressure level was adjusted to comfort- 
able hearing level in a quiet office environment. The resulting 
level was in the range 65-80 dB SPL. Out of the 10% deviant /ka/ 
stimuli, half were presented at the left and half at the right side. 
Deviants were random with different randomization sequences 
across sessions and subjects. Subjects were instructed to ignore 
the sounds and only pay attention to the visual stimuli (see 
Figure IB). 

Visual stimulation 

For visual stimulation schematic faces were chosen which exhib- 
ited either a positive or a negative expression. The positive expres- 
sion included a half-elliptic mouth shape and round-shaped 
eyebrows. The negative expression was achieved by inverting the 
mouth downwards and orienting the inner eyebrow ends upwards 
(see Figure 1A). Emotional appraisal of the stimuli was validated 
beforehand. In a pre-study on 10 healthy subjects, the rather 
unambiguous positive and negative stimuli (see Figure 1A) were 
rated as happy or sad, respectively, without variance. 

Visual stimuli were presented with an SOA of 175 ms after 50% 
of the deviants with a stimulus duration of 150 ms. Visual stimuli 
were either of positive or negative valence and appeared either in 
the left or right visual field, spatially congruent or incongruent 
with the deviant syllable. Participants were instructed to press a 
button with the index finger of the right hand as fast as possible at 
stimulus appearance without distinction of visual field or valence. 
Subjects were instructed to fixate on a cross in the middle of the 
screen. 

PROCEDURE 

Computing the factors valence (positive, negative), visual field 
(left, right), and spatial congruency with deviant (left-ear deviant, 
right-ear deviant), eight (2x2x2) conditions resulted for 
acoustic deviants followed by a visual stimulus. As control condi- 
tion, standard events followed by a visual stimulus were presented 
as well, providing four different conditions (two 2-level factors 
valence and visual field). The experiment comprised 15 stimuli 
for each of the eight "deviant + face" conditions and 30 stim- 
uli for each of the four "standard + face" conditions to balance 
the number of stimuli. The design also accounted for a bal- 
anced distribution of deviant acoustic stimuli and visual stimuli 
throughout the oddball sequence with a minimum of two stan- 
dard syllable pairs (/baL/-/baR7) in between syllable pairs with 
a deviant (/baL/-/ka/ or /ka/-/baR/). Moreover, a minimum of 
two "no face trials" was maintained between two "face trials" to 
prevent an overlap of the button press in response to a visual 
stimulus and the presentation of the next visual stimulus. In the 
randomization procedure, first, the auditory oddball design was 
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randomized; second, randomly 15 left and 15 right deviant events 
were selected and assigned to the "deviant + face" condition. 
Third, from the standard events one was randomly selected and 
assigned to the "standard + face" condition only if it maintained 
at least two-trial distance from any other face trial; this proce- 
dure was repeated until 30 trials were successfully assigned to the 
"standard + face" condition. 

Stimulus presentation with ms-precise synchronization 
between auditory and visual modality and response time 
recording was conducted using the software Presentation 10.0 
(Neurobehavioral Systems, Inc., Albany, CA) using a standard 
personal computer. The experiment lasted about 20 min for each 
participant. 

DATA ANALYSIS 

Button presses in response to visual stimuli were analyzed by sub- 
tracting onset of the visual stimulus from onset of button press. 
The mean reaction time value (in ms) was obtained for each par- 
ticipant. The reaction to visual stimuli after standard events was 
compared to the reaction to visual stimuli after deviant events 
without considering spatial congruency in order to detect possible 
alerting effects of the deviant syllables; therefore, responses of the 
two valence and the two visual field conditions were pooled in the 
analysis (T67). A 2 x 2 x 2 repeated-measures ANOVA with the 
factors valence (positive, negative), visual field (left, right), and 
spatial congruency (left-ear deviant, right-ear deviant) was con- 
ducted. Subject was the repeating factor. Significance level was set 
at p < 0.05. Paired t-tests disentangled the effects post-hoc. 

RESULTS 

Overall deviant stimuli had a facilitating effect on responses to 
succeeding face stimuli as compared to standard stimuli (Tg7 = 
—5.369, p < 0.001); this acceleration was further modulated by 



stimulus properties: the repeated-measures ANOVA analyzing the 
influence of the factors valence of the visual stimulus, visual field 
and spatial congruency with acoustic deviant on reaction time 
yielded no significant main effect but significant interactions of 
valence and visual field 16) = 6.122, p = 0.025] as well as 
valence and spatial congruency [F(i i \g) = 9.595, p = 0.007]. The 
other double and triple interactions failed significance (all p > 
0.15). To account for a potentially skewed distribution of the 
reaction times, the analysis was repeated after square root trans- 
formation; no influence of the distribution emerged (valence X 
visual field: F(i_ 16) = 6.753, p = 0.019; valence X spatial con- 
gruency: F(i_ 16) = 7.728, p = 0.013; other p > 0.15). Pair-wise 
comparisons revealed significantly faster responses to positive 
stimuli in the left visual field after left-ear deviants compared 
to after right-ear deviants, reflecting a reaction time accel- 
eration due to spatial congruency (Ti6 = —4.024, p = 0.001; 
Figure 2A). Spatial incongruence did not provoke a similar slow- 
ing in the right visual field: responses to incongruent positive 
stimuli in the right visual field were significantly faster than 
those in the left visual field (T 16 = 2.338, p = 0.033; Figure 2A). 
No significant effect emerged when stimuli expressed nega- 
tive valence (Figure 2B). Non-parametric testing confirmed both 
post-hoc Mests (Wilcoxon-Test: W = 141, p < 0.001; W = 117, 
p < 0.028), which may argue against an outlier effect in the 
relatively small sample. 

DISCUSSION 

In the present study, acoustic deviants in a dichotic oddball 
paradigm were expected to direct attention across modalities 
in interaction with emotions. In analogy to the cueing effect 
of Posner, deviants were hypothesized to serve as crossmodal 
spatial cues — with positive and negative schematic faces in the 
left or right visual field as targets. No overall congruency effect 



Frontiers in Behavioral Neuroscience 



www.frontiersin.org 



February 2013 | Volume 7 | Article 6 | 3 



Schock et al. 



Auditory deviants in spatial attention 




FIGURE 2 | Significant crossmodal interactions of stimulation sides 
emerged in the (A) positive valence condition but not in the (B) negative 
valence condition. A congruency effect could be detected in the left visual 



field with positive valence, whereas the detection of stimuli with negative 
valence was not speeded by cross-modal congruency (*p < 0.05; **p < 0.01; 
mean ± SE). 



emerged, but an interaction of emotional valence and congruency 
effect emerged. Reactions in response to positive visual stimuli in 
the left hemifield were slower when deviants had shifted atten- 
tion to the right beforehand, compared to responses to spatially 
congruent stimuli. Specifically, visual stimuli expressing negative 
emotion enabled faster reaction due to attention capture. 

ATTENTION-DIRECTING DEVIANTS 

The mismatch response is a well-established tool in the investiga- 
tion of neural responses expressing relevance and discrimination 
of stimulus features (Naatanen and Winkler, 1999). It is thought 
to measure attention mechanisms without the explicit allocation 
of attention (Naatanen, 1992). Indeed, our study confirmed that 
the presentation of a deviant stimulus speeded the responses to 
visual stimuli. This is in line with previous studies on cueing (see 
Posner, 1980) and designs comparable to the unattended odd- 
ball design (e.g., Sussman et al, 2003). Spatial resource allocation 
was reflected in designs presenting different stimuli to the left and 
the right ear. Deviant acoustic stimuli direct attention to one side 
resulting in slower processing at the other side (Schroger, 1996). 
We studied whether acoustic deviants are suitable to cue stim- 
uli of another modality in different spatial locations. The cueing 
paradigm described by Posner (1980) considered valid peripheral 
cues indicating the location of the target and invalid cues directing 
attention away from the target location. Time-dependent relation 
of cue and target is a critical factor to maintain relevance of the 
cue for the target detection, i.e., below 300 ms (compare Spence 
and Driver, 1994). We found a spatially selective slowing of the 
detection of incongruent targets, i.e., positive expressions on the 
opposite side to the preceding deviant. This confirms that the 
processing of deviant stimuli — even if unattended — may lead to 
shifts in supramodal spatial attention. 

Previously we have found that mismatch responses to dichotic 
deviants are modulated by lateralized stimuli in other modalities 
(Mathiak et al, 2005). The responses to deviants were enhanced 
by visual stimuli at or toward the same side. In the same vein, 
tactile congruent stimuli increased responses to auditory events 



as early as 100 ms after stimulus onset (Menning et al, 2005). 
Taken together these findings suggest a bidirectional interaction 
between spatial attention across modalities even in the absence of 
explicit attention. In these studies, the spatial nature of the stim- 
uli emerged from dichotic presentation considering the left and 
right auditory channel as equivalent to a spatial location in the left 
and right hemifield. Therefore, spatial localization or mere hemi- 
spheric representation may underlie the observed multisensory 
interactions. 

The audio-visual interaction seems to be strongest at the right 
hemisphere. We found the incongruence effect in the left visual 
field only. This is in accordance with the effects of visual stimuli 
on mismatch responses, which were stronger at the right hemi- 
sphere (Mathiak et al, 2005). Considering the face-like target 
stimuli, this is well in-line with models of right-lateralized face 
processing (Kanwisher et al, 1997). More general, spatial atten- 
tion networks were considered lateralized to the right hemisphere 
(Corbetta and Shulman, 2002). Thus a speeding of responses to 
visual stimuli may have been expected strongest in the left hemi- 
field. Moreover, this lateralization pattern suggests interaction 
with right-hemispheric emotion processing as well. 

INTERACTION WITH EMOTIONS 

Models of hemispheric dominance of emotion processing local- 
ize positive valence in the left hemisphere and negative valence 
in the right hemisphere (for a review, see Demaree et al., 2005). 
Particularly automatic processing of emotions was found to elicit 
right-lateralized patterns (e.g., Dyck et al, 2011). Mismatch nega- 
tivity has been found to be reduced by visual stimuli with positive 
but not negative valence (Surakka et al., 1998). In our study, 
reaction time costs in response to spatially incongruent stimuli 
emerged only in the positive valence condition (Figure 2) as well. 
This may be attributed to low arousal values of sadness. In con- 
trast considering high arousal, positive and negative emotions 
(happy and angry) ameliorated lateralized attention deficits in 
hemineglect; Vuilleumier and Schwartz (2001) found schematic 
face to bias attention to one side more strongly than neutral 
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faces even when conscious processing was impaired. Amygdala 
and extrastriate cortex were thought to be the underlying neu- 
ral networks (Vuilleumier et al., 2002). Similar to faces, emo- 
tional prosody yielded attention shifts into the affected hemispace 
(Grandjean et al., 2008). Emotional arousal and spatial attention 
conceivably interact at early processing stages (see Grabowska 
etal.,2011). 

Negative valence — even if not task-relevant and not 
attended — may suffice to interfere with a right-hemispheric 
network that enables crossmodal shifts of attention. Indeed, 
mood induction may also modulate mismatch responses at the 
right hemisphere (Schock et al, 2012). Thus, similar to the 
bidirectional interaction of spatial attention across modalities, 
bidirectional interaction between spatial attention and negative 
emotion can be expected. Indeed, altered mood in depression can 
lead to a bias in visuospatial attention (Liotti and Mayberg, 2001), 
which maybe due to reduced right-hemispheric arousal (Schock 
et al., 2011). It is rather notable how apparently small context 
effects can influence crossmodal interactions. 

As an alternative to the valence hypothesis, the right- 
hemisphere theory hypothesizes dominance of the right hemi- 
sphere in processing emotions (Borod et al., 1998). Therefore, 
negative valence may be expected to be unaffected by reaction 
time costs because of higher emotional salience of the stimuli 
(Eastwood et al., 2001; Fenske and Eastwood, 2003). The mech- 
anism may not differ completely since the predicted effects on 
the attention network are similar (Killgore and Yurgelun-Todd, 
2007). Indeed, more salient positive stimuli can be expected to 
reduce the crossmodal interaction influencing detection speed. 

LIMITATIONS 

One feature of mismatch experiments is that the number of 
target stimuli is relatively low, particularly if they are dis- 
tributed between different conditions. Therefore, outliers may 



have affected the statistical analysis despite corroborating evi- 
dence from root-transformed data and non-parametric testing. 
Extending the test duration much over the 20min may have 
unwanted effects on attention and arousal. Naatanen et al. (2004) 
suggested a variant with interleaved deviants with different audi- 
tory features. Such a design however is not feasible with the inter- 
vening visual stimuli. Furthermore, the number of subjects in the 
present experiment (n = 17) is not high enough for conclusions 
on generalizability. For instance the schematic faces were previ- 
ously applied successfully to study valence effects (e.g., Eastwood 
et al, 2001) but the generalizability to less abstract depictions or 
natural faces is unclear (compare Dyck et al., 2010). A number of 
additional confounders still need to be taken into account such as 
the type of the auditory stimuli (phonetic or noises), the timing of 
the stimuli, the level of alertness, and the task (simple detection, 
localization, or discrimination). Therefore, another limitation lies 
in the high number of parameters which may influence the level 
of interaction between the modalities. 

CONCLUSION 

Reaction times confirmed crossmodal spatial cueing effects of 
auditory deviants in a mismatch design. Deviant stimuli in a 
task- irrelevant oddball design were shown to serve as attention- 
directing cues that are effective across modalities. Negative 
valence processing in the right hemisphere and emotional salience 
reduce susceptibility to these cueing effects. 
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